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Abstract 

The game dynamical equations are derived from Boltzmann-like equations for individual 
pair interactions by assuming a certain kind of imitation behavior, the so-called propor- 
tional imitation rule. They can be extended to a stochastic formulation of evolutionary 
game theory which allows the derivation of approximate and corrected mean value and 
covariance equations. It is shown that, in the case of phase transitions (i.e. multi-modal 
probability distributions), the mean value equations do not agree with the game dynamical 
equations. Therefore, their exact meaning is carefully discussed. Finally, some generaliza- 
tions of the behavioral model are presented, including effects of expectations, other kinds 
of interactions, several subpopulations, or memory effects. 



1 Introduction 

Since von Neumann and Morgenstern have initiated the field of game theory,^ it has often 
proved of great value for the quantitative description and understanding of the competition 
and co-operation between individuals. Game theory focusses on two questions: 1. Which 
is the optimal strategy in a given situation? 2. What is the dynamics of strategy choices in 
cases of repeatedly interacting individuals? In this connection game dynamical equations^] 
find a steadily increasing interest. Although they agree with the replicator equations of 
evolution theory (cf. Sec. 2), they cannot be substantiated in the same way. Therefore, 
we will be looking for a foundation of the game dynamical equations which bases on 
individual actions and decisions (cf. Sec. 4). 

In addition, we will formulate a stochastic version of evolutionary game theory (cf. Sec. 
3). This allows to investigate the effects of fluctuations on the dynamics of social systems. 



^ J. von Neumann and O. Morgenstern, Theory of Games and Economic Behavior. Princeton: Princeton 
University Press 1944. 

^P. Taylor and L. Jonker, "Evolutionarily Stable Strategies and Game Dynamics", in: Math. Bio- 
sciences 40, 1978, pp. 145-156. 

J. Hofbauer and K. Sigmund, Evolutionstheorie und dynamische Systeme. Berlin: Parey 1984. 
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In order to illustrate the essential ideas, a concrete model for the self-organization of 
behavioral conventions is presented (cf. Sec. 5). We will see that the game dynamical 
equations describe the average evolution of social systems only for a certain time period. 
Therefore, a criterium for their validity will be developed (cf. Sec. 6). Finally, we will 
present possible extensions to more general behavioral models and discuss the actual 
meaning of the game dynamical equations (cf. Sec. 7). 



2 The Game Dynamical Equations 

Let Px(t) with 

0<Px{t)<l and 5]p,(t) = l (1) 

X 

denote the proportion of individuals pursuing the behavioral strategy x E S at time t. We 
assume the considered strategies to be mutually exclusive. The set 5* of strategies may 
be discrete or continuous, finite or infinite. The only difference will be that sums over 
X are to be replaced by integrals in cases of continuous sets. By A^y we will denote the 
possibly time-dependent payoff of an individual using strategy x when confronted with 
an individual pursuing strategy y. Hence, his/her expected success {Ex)t will be given by 
the weighted mean value 

{E.;)t = Y.AxyPy{t), (2) 

y 

since Py is the probability that the interaction partner uses strategy y. In addition, the 
average expected success will be 



{E)t = J2p.it){E,)t = T.EpAt)AxyPyit) . (3) 

X X y 

Assuming that the relative temporal increase {dpx/dt)/px of the proportion px of individ- 
uals pursuing strategy x is proportional to the difference between the expected success 
{Ex)t and the average expected success {E)t, we obtain the game dynamical equations 

"^^^^^^ = i^PximExh-Wt] 

= upx{t)\{Ex)t-Y.Pyit)iEy)t\, (4) 



dt 



where the possibly time-dependent proportionality factor u is a. measure for the interac- 
tion rate with other individuals. According to (|), the proportions of strategies with an 
above-average success {Ex)t > {E)t increase, whereas the other strategies will be dimin- 
ished. Note, that the proportion of a strategy does not necessarily increase or decrease 
monotonically. Certain payoffs are related with an oscillatory or even chaotic dynamic^. 

Equations (^) are identical with the replicator equations from evolutionary biology. They 
can be extended to the selection-mutation equations 

dpxit) 



dt 



= upx{t)[{Ex)t-Y.Pyit){Ey)i 
y 

+ H [P?/(^)wi(2/ ^ a;) -p^(t)wi(x y)] . (5) 



•^W. Schnabl, P. F. Stadler, C. Forst, and P. Schuster, "Full Characterization of a Strage Attractor. 
Chaotic Dynamics in Low-Dimensional Replicator Systems", in: Physica D 48, 1991, pp. 65-90. 



The terms which agree with describe a selection of superior strategies. The new terms 
correspond to the effect of mutations, i.e. to spontaneous changes from strategy x to 
other strategies y with possibly time-dependent transition rates Wi{x — > y) (last term) 
and the inverse transitions. They allow to describe trial and error behavior or behavioral 
fluctuations. 



3 Stochastic Dynamics: The Master Equation 

Let us consider a social system consisting of a constant number 

N = J2nAt) (6) 

X 

of individuals. Herein, n^{t) denotes the number of individuals who pursue strategy x at 
time t. Hence, the time-dependent vector 

n= {ni,n2,...,n^,...,ny,...) (7) 

reflects the strategy distribution in the social system and is called the socioconfiguration. 
If the individual strategy changes are subject to random fluctuations (e.g. due to trial 
and error behavior or decisions under uncertainty), we will have a stochastic dynamics. 
Therefore, given a certain socioconfiguration uq at time t^, for the occurence of the strategy 
distribution n at a time t > t^ we can only calculate a certain probability P{n,t). Its 
temporal change dP/ dt is governed by the so-called master equatior^ 

^^^^ = E [^(^'' tWi^' ^n)- P{n, t)W{n ^ n')] . (8) 

n' 

The sum over -n' extends over all socioconfigurations fulfilling Ux € {0, 1,2,.. .} and (^. 

According to equation (P), an increase of the probability P{n,t) of having socioconfigu- 
ration n is caused by transitions from other socioconfigurations n' to n. While a decrease 
of P{n,t) is related to changes from n to other socioconfigurations n'. The correspond- 
ing changing rates are proportional to the configurational transition rates W{n "n') of 
changes to socioconfigurations n' given the socioconfiguration n and to the probability 
P(n, t) of having socioconfiguration n at time t. 

The configurational transition rates W have the meaning of transition probabilities per 
time unit and must be non-negative quantities. Frequently, the individuals can be assumed 
to change their strategies independently of each other. Then, the configurational transition 
rates have the form 

Win ^n') = I ^^^(^ ~^ ^' ^ fg) 

1 otherwise, 

i.e. they are proportional to the number of individuals who may change their strategy 
from X to another strategy y with an individual transition rate w{x ^ y;n) > 0. In 
relation (^), the abbreviation 

fixy = {ni,n2, . . . - I, . . . ,ny + 1, . . .) (10) 



"'W. Wcidlich and G. Haag, Concepts and Models of a Quantitative Sociology. The Dynamics of In- 
teracting Populations. Berlin: Springer, 1983. 
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means the socioconfiguration which results after an individual has changed his/her strat- 
egy from X to y. 

It can be shown that the master equation has the properties 

P(n,t)>0 and ^P(n,t) = l (11) 

n 

for all times t, if they are fulfilled at some initial time to- Therefore, the master equation 
actually describes the temporal evolution of a probability distribution. 



4 Approximate Mean Value Equations 

In order to connect the stochastic model to the game dynamical equations, we must specify 
the individual transition rates w in a suitable way. Therefore, we derive the mean value 
equations related to the master equation (^ and compare them to the selection-mutation 
equations (H). 

The proportion is defined as the mean value 

U)t = Y.i\^^t)nn,t) (12) 

of the number /(n, t) = of individuals pursuing strategy x, divided by the total number 
of considered individuals: 

P.W = ^ = ^E^x.P(rI,t). (13) 

n 

Taking the time derivative of {nx)t and inserting the master equation gives 

d{n^)t 



dt 



Y,n^[P{n\ t)W{n ^ n) - P{n, t)W{n n')] 

n 

J2K-n,)W{n^n')P{n,t), (14) 



where we have interchanged n and n' in the first term on the right hand side. Taking into 
account relation (^, we get 

= '^[nyw{y ^ x;n) - n^w{x y;n)]P{n,t) . (15) 



With (13) this finally leads to the approximate mean value equations 
dpx{t) 



dt y 



J2 [Pyi^)^iy (^)t) - Px{t)w{x y; {n)t)] (16) 



However, these are only exact, if the individual transition rates w are independent of 
the socioconfiguration n. Anyhow, they are approximately valid as long as the probability 
distribution P{n,t) is narrow, so that the mean value {f{n,t))t of a function f{n,t) can 
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be replaced by the function f{{n)t,t) of the mean value. This problem will be discussed 
in detail later on. 

Comparing the rate equations (|l^) with the selection-mutation equations we find a 
complete correspondence for the case 



w{y ^ x;n) =wi{y ^ x) +W2{y x)nx (17) 



with 

and the success 



W2{y ^ ^) = max{E^ - By, 0) (18) 



Ti 

E. = Y.A^y^^ (19) 

y 



smce 

max((E,)t - {Ey)t, 0) - max((Ey)t - {E,)t, 0) = {E,)t - {Ey)t . (20) 

Whereas Wi is again the mutation rate (i.e. the rate of spontaneous transitions), the 
additional term in (p!7| ) describes imitation processes, where individuals take over the 
strategy x of their respective interaction partner. Imitation processes correspond to pair 
interactions of the form 

y + X ^ X + X . (21) 

Their frequency is proportional to the number Ux of interaction partners who may convince 
an individual of strategy x. The proportionality factor W2 is the imitation rate. 

Relation (|TB|) is called the proportional imitation rule and can be shown to be the best 
learning rule.0 It was discovered in 19920 and says that an imitation behavior only takes 
place, if the strategy x of the interaction partner turns out to have a greater success E^^ 
than the own strategy y. In such cases, the imitation rate is proportional to the difference 
[E^ — Ey) between the success' of the alternative x and the previous strategy y, i.e. 
strategy changes occur more often the greater the advantage of the new strategy x would 
be. 

All specifications of the type 

W2iy -.x) = C+ ^[XE, - (1 - X)Ey] (22) 

with an arbitrary parameter A also lead to the game dynamical equations. However, 
individuals would then, with a certain rate, take over the strategy x of the interaction 
partner, even if its success E^ is smaller than that of the previously used strategy y. 
Moreover, if C is not chosen sufficiently large, the individual transition rates w > can 
become negative. 

In summary, we have found a microscopic foundation of evolutionary game theory which 
bases on four plausible assumptions: 1. Individuals evaluate the success of a strategy 
as its average payoff in interactions with other individuals (cf. ([T9|)). 2. They compare 
the success of their strategy with that of the respective interaction partner, basing on 
observations or an exchange of experiences. 3. Individuals imitate each others behavior. 
4. In doing so, they apply the proportional imitation rule (|18D [or (^2]) ]. 

^K. H. Schlag, "Why Imitate, and if so, How? A Bounded Rational Approach to Multi- Armed Bandits" , 
Discussion Paper No. B-361, Department of Economics, University of Bonn. 

^D. Hclbing, "Interrelations between Stochastic Equations for Systems with Pair Interactions", in 
Physica A 181, 1992, pp. 29-52. 



5 Self-Organization of Behavioral Conventions 



For illustrative reasons, we will now discuss an example which allows to understand how 
social conventions emerge. We consider the simple case of two alternative strategies x G 
{1,2} and assume them to be equivalent so that the payoff matrix is symmetrical: 



A + B B 
B A + B 



(23) 



If y4 > 0, the additional payoff A reflects the advantage of using the same strategy like the 
respective interaction partner. This situation is, for example, given in cases of network 
externalities like in the historical rivalry between the video systems VHS and BETA 
MAXQ. Finally, the mutation rates are taken constant, i.e. Wi{x y) = Wi. 

The resulting game dynamical equations are 



dpxjt) 
dt 



-2 



Px{t) - 



{Wi + uAp^{t)[p^{t) 



1 



Obviously, they have only one stable stationary solution if the (control) parameter 

4Wi 



K 



uA 



is smaller than zero. However, for > equation (|2^) can be rewritten in the form 



dpxjt) 
dt 



-2uA 



Pxit) - - 



Px{t) 



1 + v^' 



Px{t) - 



(24) 



(25) 



(26) 



The stationary solution p^; = 1/2 is unstable, then, but we have two new stable stationary 
solutions Px = (1/2 ± ^/K/2). That is, dependent on the detailled initial condition, one 
strategy will win the majority of users although both strategies are completely equivalent. 
This phenomenon is called symmetry breaking. It will be suppressed, if the mutation rate 
Wi is larger than the advantage effect uA/A. 

The above model allows to understand how behavioral conventions come about. Examples 
are the pedestrians' preference for the right-hand side (in Europe), the revolution direction 
of clock hands, the direction of writing, or the already mentioned triumph of the video 
system VHS over BETA MAX. 

It is very interesting how the above mentioned symmetry breaking affects the probability 
distribution P{n,t) = P{ni,n2,t) = P{ni,N — ni,t) of the related stochastic model 
(cf. Fig. |IQ). For K < the probability distribution is located around ni = N/2 = n2 
and stays small so that the approximate mean value equations are applicable. At the 
so-called critical point k = 0, a phase transition to a qualitative different system behavior 
occurs and the probability distribution becomes very broad. As a consequence, the game 
dynamical equations do not correctly describe the temporal evolution of the mean strategy 
distribution anymore. 

^W. B. Arthur, "Competing Technologies, Increasing Returns, and Lock-In by Historical Events", in 
The Economic Journal 99, 1989, pp. 116-131. 

^For illustrative reasons, a small number of individuals {N — 40) and a broad initial probability 
distribution have been chosen. In each picture, the box is twice as high as the maximal occuring value of 
the probability. 



For K > 0, a bimodal and symmetrical probability distribution evolves. That is, the 
likelihood that one of the two equivalent strategies will win through is much larger than 
the likelihood to find approximately equal proportions of both strategies. At the beginning, 
the initial state or maybe some random fiuctuation determines, which strategy has better 
chances to win. However, in the long run both strategies have exactly the same chance. 
It is clear, that in such cases the game dynamical equations fail to describe the mean 
system behavior (cf. Fig. Q), which would correspond to the average temporal evolution 
of an ensemble of identical social systems. In cases of oscillatory or chaotic solutions of 
the game dynamical equations the situation is even worse. 



6 Exact, Approximate, and Corrected Mean Values 
and Variances 

In the last section we have seen that the approximate mean value equations 

^ = MMn)t) (27) 
with the so-called first jump moments 

M,{n) = ^« - n,)W{n ^ n') (28) 



(cf. (|T4D) are not sufficient. This calls for corrected mean value equations and a criterium 
for the time period of their validity. If the individual transition rates w{x —>■ y; n) depend 
on the socio configuration, the exact mean value can only be evaluated via formula (p^. 
This requires the calculation of the probability distribution P{n,t) and, therefore, the 
numerical solution of the respective master equation (^. Since the number of possible 
socioconfigurations is normally very large, an extreme amount of computer time would 
be necessary for this. 

Luckily, it is possible to derive from (|T^) the corrected mean value equations 



by means of a suitable Taylor approximation. This equation depends on the covariances 
(yxy{t) = (^{n^ - {nx)t){ny - {ny)t))^ = ^{nx- {nx)t){ny - {ny)t)P{n,t) (30) 

it 

which can be determined by means of the covariance equations 



^ r f)i\/T J 



(31) 



The functions 

Mxy{n) = EK - nx)in'y - n,)W^(n ^ n') (32) 
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are called the second jump moments. 

Equations (^9|) and ( ^Tl) build a closed system of equations, but still no exact one, since 
this would depend on higher moments of the form {nxHyUz ■ ■ ■)t- Nevertheless, according 
to Figure |^ the corrected mean value equations yield significantly better results than 
the approximate ones. As a consequence, they are valid for a much longer time period. 
Suitable validity criteria are the relative variances 

V;(t):=^f^, (33) 

since these are a measure for the relative width of the probability distribution P{n,t). 
It can be shown that the covariances and all higher moments are small, if only Kr(t) is 
much smaller than 1 for every x. Numerical investigations indicate that the approximate 
mean value equations begin to separate from the exact ones as soon as one of the rela- 
tive variances Vx{t) becomes greater than 0.04. The corrected mean value equations and 
covariances remain reliable as long as Kc(t) is smaller than 0.12 for all x (cf. Fig. 

A more detailled discussion of the above matter is presented elsewhere]^. 



7 Diverse Generalizations 



The above discussed behavioral model can be generalized in different respects. 



Modified transition rates: The strange cusp at ni = N/2 in Figure ^ which comes 
from the discontinuous derivative of W2{x y) at = Ey, can be avoided by the 



modified imitation rates 



z/ exp{Ex - Eyj 



W2{y^x) = -^^ ^ with Dxy = Dyx = 2. (34) 

This ansatz agrees with relation (^) in linear approximation for C = v/ {2N) and A = 1/2, 
but it always yields non-negative imitation rates. Similar to (|T8|) it guarantees two essential 
things: 1. The imitation rate grows with an increasing gain {E^ — Ey) of success. 2. If the 
alternative strategy x is inferior, the imitation rate is very small (but, due to uncertainty, 
not negligible). The results of the corresponding stochastic behavioral model are presented 
in Figure They show the usual flatness of the probability distribution P{ni, N — ni,t) 
at the critical point k = 0, where again a phase transition occurs. 



Dynamics with expectations: The decisions of individuals are often influenced by 
their expectations {Ex)li about the success of a strategy x at future times t' > t. These 
will base on some kind of extrapolation of past experiences with the success of x. If 
expected payoffs at future times t' are weighted exponentially with their distance (t' — t) 
from the present time t, one would setQ 

oo / f \ 

{Ex)t = tI^^' [-Y^j ■ 

^D. Helbing, "A Stochastic Behavioral Model and a 'Microscopic' Foundation of Evolutionary Game 
Theory", in Theory and Decision 40, 1996, pp. 149-179. 

^°N. S. Glance and B. A. Huberman, "Dynamics with Expectations", in Physics Letters A 165, 1992, 
pp. 432-440. 
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Other kinds of pair interactions: Apart from imitative behavior, individuals also 
sometimes show an avoidance behavior 

X + X ^ y + X , (36) 

especially if they dislike their interaction partner (so-called 'snob effect'). This can be 
taken into account by an additonal contribution to the individual interaction rates: 

w{y x;n) = wi{y x) + W2{y x) + njz{y x)ny. (37) 

W3 denotes the avoidance rate. 



Several subpopulations: Sometimes one has to distinguish different subpopulations a, 
i.e. different kinds of individuals. This is necessary, if not all individuals have the same 
set S of strategies.[^ A similar thing holds, if the considered social system consists of 
competing groups, where only individuals of the same group behave cooperatively. The 
generalized behavioral equations areQ 

^ = E Hit)w^iy - {n)t) - Pl{t)w\x y; {n),)] (38) 

with individual interaction rates of the form 

w\y -^x-n)= wl{y x) + ^ [wf{y ^ x) n\ + wt{y ^ x) n\\ . (39) 

h 



Inclusion of memory effects: If the strategy distribution at past times t' < t influ- 
ences present decisions in a non-Markovian way, the approximate mean value equations 
have the form 

^ = E / m>t-Ay - ^; {n),) - pl{t')wU{x -> y- {n),)] . (40) 

y —00 

For example, in cases of an exponentially decaying memory one would have 

w1_t,{x y; {n)t') = w^ix y; {n)t>) ^ exp [^^^j • (41) 



8 Summary and Conclusions 



We have found a microscopic foundation of the game dynamical equations, basing on a 
certain kind of imitative behavior. Moreover, a stochastic version of evolutionary game 
theory has been formulated. It allowed to understand the self-organization of social con- 
ventions as a phase transition which is related with symmetry breaking. Moreover, we have 

J. Hofbauer and K. Sigmund, The Theory of Evolution and Dynamical Systems. Cambridge: Cam- 
bridge University Press 1988. 

^^D. Helbing, Quantitative Sociodynamics. Stochastic Methods and Models of Social Interaction Pro- 
cesses. Dordrecht: Kluwer Academic 1995. 



n 



seen that the game dynamical equations correspond to approximate mean value equations. 
Normally, they agree with the mean value equations of stochastic game theory for a cer- 
tain time period only, which can be determined by calculating the relative variances. For 
an improved description of the average system behavior we have derived corrected mean 
value equations which require the solution of additional covariance equations. 

The interpretation of the game dynamical equations follows by reformulating these in 
terms of a social force modeljQ assuming a continuous strategy set: 

dxa{t) ^ f^^^^-^ _^ f2{xa,xp) + fluctuations. (42) 

The force term 

fl{,Xa) = J dx {x — Xa) Wi{Xa x) (43) 

delineates spontaneous strategy changes by individual a, whereas 

f2{Xa,Xf3) = (XfS - Xa)w2{Xa ^ Xp) 

+ J dx {x — Xa) Ws^Xa — > x) S{Xa — Xp) (44) 

is the interaction force which originates from individual (3 and influences individual a. 
Here, 6{x~y) denotes Dirac's delta function (which yields a contribution for x = y only). 
According to (^21), the game dynamical equations describe the most probable strategy 
changes rather than the average (representative) evolution of a social system. Therefore, 
they neglect the effects of fluctuations on the system behavior. 

A more detailled discussion of the results presented in this paper is available elsewhere^^'^^. 



^D. Helbing, Stochastische Methoden, nichtlineare Dynamik und quantitative Modelle sozialer Prozesse, 
2nd edition. Aachen: Shaker 1996. 
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Figure 1: Probability distribution P{n,t) = P{ni,N — ni;t) of the socioconfiguration n 
for varying values of the control parameter k according to the stochastic version of the 
game dynamical equations. 
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Figure 2: The numerical solutions of the approximate mean value equations (• • •) agree 
with those of the exact mean value equations ( — ) only for a short time interval. The 
corrected mean value equations ( — ) yield much better results, although they also deviate 

from the exact curves when the relative variances ( ) become too large. Nevertheless, 

they describe the average long-term behavior properly. 
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Figure 3: Probability distribution P{n,t) = P{ni,N — ni;t) of the socioconfiguration n 
according to the modified stochastic game dynamical equations. 



